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© Scene based nonuniform^ correction for scanning infrared sensors. 



© A scene based nonuniformity correction method 
(40) that computes and applies offset correction er- 
rors to a video signal corresponding to an image 
derived from a imaging sensor (11). A video signal 
derived from the sensor (11) is processed such that 
a vector representing offset correction terms is 
formed, and this vector is initially set to zero. Each 
element in this vector represents a correction term 
for a particular detector of the sensor (11). The 
vector is applied to each pixel of the image by a 
processor (13) as the pixels are read from the sen- 
sor (11). To measure the offset error, the image is 
separated into vertically oriented regions, each com- 
prising a plurality of channels. The average of each 
channel within a region is computed (42), and a set 
of region vectors is formed, such that there is one 
region vector for each region. Each region vector is 
then globally high-pass filtered and then edges larg- 



er than a predefined threshold are detected (43), and 
marked (44). Then, each region vector is further 
separated into sub-regions (45). The isolated sub- 
regions are locally high-pass filtered. In one embodi- 
ment, the correction terms for each vertical region 
vector are averaged together, resulting in a single 
correction vector (48). The correction terms calcu- 
lated for each vertical region may also be applied 
individually to each detector of the sensor (11). In 
this second embodiment, the offset level error in 
each region for each channel is calculated (49), 
wherein the offset level error at boundary edges is 
undefined- The correction terms corresponding to a 
region are applied as the detector (11) scans the 
scene and views a portion corresponding to that 
particular region. The correction terms are smoothed 
at region boundaries to eliminate noise due to 
boundary transitions. 



Rank Xerox (UK) Business Services 

(3.10/3.09/3.3-41 



EP 0 653 882 A1 



11 



12 



SCENE - 



DETECTOR READOUT 



INPUT : 



x (rn, n)| 



10^ 



Fig. 1 



21 22 23 24 



16 

t 



COARSE 
LEVEL 



28 

[ MEMORVl 



13-^: 



28 

IMEMORVI 



COARSE 






FINE 


GAIN 




18 

/ 


GAIN 



FINE 
LEVEL 




OUTPUT: 



y <m. ") 



20 



NONUNIFORMITY 
ESTIMATOR 



memoryI 
— 



25 28 



htp X 4o; 



2 



V 



EP 0 653 882 A1 



BACKGROUND 

The present invention relates generally for im- 
aging sensors, and more particularly, to scene 
based nonunrformity correction methods for use 
with such imaging sensors. 

Nonuniformities appear at an output display of 
an imaging sensor as fixed pattern noise. The 
nonuniformities are described as noise because 
they result in undesirable information. The non- 
uniformities are described as a fixed pattern be- 
cause their characteristics do not change (or 
change relatively slowly) with time. These non- 
uniformities may also be thought of as detector 
gain and offset errors. In the method of the present 
invention, all errors are treated as offset errors. 
Thus, the present invention accurately measures 
the detector offsets using actual scene information. 

Once the offset errors have been measured, 
there are several ways in which the corrections 
may be applied. They may be used as the only 
source of error correction. They may also be used 
as fine offset correction terms, in conjunction with 
coarse offset terms and gain correction terms. 
These other terms may be calculated using a num- 
ber of different methods. These methods include 
coarse offset terms calculated using a thermal ref- 
erence source; coarse offset and gain terms cal- 
culated as part of system initialization; and fine 
gain terms calculated using thermal reference sour- 
ces or scene-based methods. 

In one current method employed by the as- 
signee of the present invention, one or more ther- 
mal reference sources are used to measure non- 
uniformities for a scanning infrared sensor and pro- 
vide data for the calculation of correction coeffi- 
cients that are employed to correct for the non- 
uniformities. There are several disadvantages re- 
lated to the use of a thermal reference source- 
based correction system First, there is added me- 
. chanical complexity which leads to increased sys- 
tem cost. Secondly, system performance may suf- 
fer. 

System performance suffers for two reasons, in 
many cases, a separate optical path is utilized for 
each thermal reference source. Thus, the correc- 
tion coefficients calculated using the thermal refer- 
ence source optical path may not be the proper 
ones for the optical path of the scanning infared 
sensor. This leads to imperfect correction. In less 
sophisticated systems, the temperature of the ther- 
mal reference source cannot be controlled. In this 
case, the thermal reference source may not be at 
the same temperature as the scene that is viewed. 
The correction coefficients thus correspond to the 
wrong part of the detector response curve. This 
also leads to imperfect correction. The present 
method avoids these problems by using scene 



temperature information. Furthermore, the present 
invention does not degrade the scene in any man- 
ner. 

5 SUMMARY OF THE INVENTION 

The present scene-based nonuniformity correc- 
tion method is used to eliminate image defects in 
an imaging sensor or video system, such as a 
to scanning infrared sensor or pushbroom sensor, for 
example, resulting from nonuniformities caused by 
an detector (focal plane array) and detector 
readout, for example. The present invention de- 
tects, measures, and corrects for nonuniformities in 
75 the Jideo output of a imaging sensor without de- 
grading the image. A set of correction terms is 
calculated and applied to a video signal produced 
by the sensor using either a feedback or feedfor- 
ward configuration. Alter the correction terms are 
20 applied, the resultant video signal is suitable for 
display or further processing. 

The video image generated by the imaging 
sensor is processed such that a vector represent- 
ing an offset correction term is formed, and this 
25 vector is initially set to zero. Each element in this 
vector represents a correction term for a particular 
detector of the imaging sensor. The vector is ap- 
plied to each pixel of the image by a processor as 
the pixels are read from the focal plane array of the 

30 sensor. 

To measure the offset error, the image is sepa- 
rated into vertically oriented regions, each compris- 
ing a plurality of channels. The average of each 
channel within a region is computed and a set of 
35 region vectors is formed, such that there is one 
region vector for each region. Each region vector is 
then processed and regions whose edges larger 
than a predefined threshold are detected and 
marked. Then, each region vector is further sepa- 
40 rated into sub-regions. The isolated sub-regions are 
high-pass filtered without regard to adjacent sub- 
regions. The high-pass filter may be in the form of 
a finite impulse response (FIR) filter or an anti- 
median filter, for example. The anti-median filter is 
45 a nonlinear high-pass filter that converges very 
rapidly. Finally, the correction terms for each verti- 
cal region vector are averaged together, resulting in 
a single correction vector that is applied to the 
video signal to reduce the effect of the nonunifor- 
50 mities. 

The correction terms calculated for each verti- 
cal region may also be applied individually to each 
the detectors. In this case, the correction terms 
corresponding to a region are applied as the detec- 
55 tor scans the scene and views a portion corre- 
sponding to that particular region. The correction 
terms are smoothed at region boundaries to elimi- 
nate noise due to boundary transitions. This sec- 
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ond method is less sensitive to gain errors in the 
detector. 

There are several advantages in using the 
present method. First the mechanical complexity of 
the imaging system is reduced, which leads to 5 
lower costs because fewer components are re- 
quired and there is reduced labor required for 
manufacturing and testing. Secondly an imaging 
system incorporating the present invention pro- 
vides better performance. 10 

BRIEF DESCRIPTION OF THE DRAWINGS 

The various features and advantages of the 
present invention may be more readily understood 75 
with reference to the following detailed description 
taken in conjunction with the accompanying draw- 
ings, wherein like reference numerals designate 
like structural elements, and in which: 

Fig. 1 shows a block diagram of a generic 20 
imaging sensor system incorporating a scene 
based nonuniformity correction method in accor- 
dance with the principles of the present inven- 
tion; 

Fig. 2 is a block diagram illustrating calculation 25 
of fine offset terms used in the scene based 
nonuniformity correction method of the present 
invention that is employed in the imaging sensor 
system of Fig. 1 ; and 

Fig. 3 is a flow diagram illustrating the scene 30 
based nonuniformity correction method in accor- 
dance with the principles of the present inven- 
tion. 

DETAILED DESCRIPTION 35 



In order to better understand the present meth- 
od or algorithm, reference is made to Fig. 1, which 
shows a block diagram of a generic scanning in- 
frared sensor system 10, or imaging system 10, 
incorporating a scene based nonuniformity correc- 
tion method 40 in accordance with the principles of 
the present invention. The scanning infrared sensor 
system 10 is comprised of a detector 11 and its 
readout 12, and the readout is coupled to system 
electronics 13 that implements scene based non- 
uniformity correction method 40. The system elec- 
tronics 13 implements correction logic that pro- 
duces coarse and fine correction terms that are 
applied to the processed video signal. The correc- 
tion logic includes two offset and gain pairs 14, 15, 
comprising a coarse offset and gain pair 14, and a 
fine offset and gain pair 15. The coarse offset and 
gain pair 14 is comprised of coarse offset level and 
coarse gain terms 16, 17 that may be calculated 
using a thermal reference source (internal or exter- 
nal) and pre-stored in a nonvolatile memory 28. 
The coarse offset level term 16 may also be cal- 



culated using 8 thermal reference source which is 
updated continuously. 

The fine offset and gain pair 15 is comprised of 
fine offset level and fine gain terms 18, 19. The fine 
gain term 19 may be set to unity, calculated using 
thermal reference sources, or calculated using a 
scene-based algorithm. First and second adders 
21, 23 and first and second multipliers 22, 24 are 
employed to appropriately combine the coarse and 
fine level and gain terms 16, 17, 18, 19 to produce 
a corrected video output signal. The present meth- 
od 40 or algorithm is used to estimate the fine level 
correction terms 18 and is performed in a non- 
uniformity estimator 20. The output of the non- 
uniformity estimator 20 has a loop attenuation fac- 
tor (k) 25 applied thereto and is coupled to a first 
input of a third adder 26. A second input of the 
third adder 26 is provided by the fine level term 18. 
As such the fine level term 18 is updated with the 
output of the nonuniformity estimator 20 multiplied 
by the loop attenuation factor (k) 25. 

The expressions and equations that are used in 
implementing the present algorithm or method 40 
will now be described. The variables associated 
with the system 10 are as follows: 

x(m, n) = input, 
y(m, n) ■ output, 

Lc(m) = coarse offset level term 1 6, 
G c (m) = coarse gain term 17, 
Lp(m, n) = fine offset level term 18, 
G F (m) s fine gain term 19, 
L(m, n) = error estimate, and 
k a loop attenuation factor 24, 

where 

m = (0, M-1); M = number of detectors, and 
n = (0, N-1); N = samples/detector. 

The system input and output are thus related by 
the equation: 

y (m, n) = G F (m) {(G c (m) [x (m, n) + Lc(m)] + Lp- 
45 (m, n)}. 

The fine level terms are recursively update after 
each frame. Thus, 

50 L F {m, n) = L F (m, n) + kL(m, n). 

Fig. 2 is a block diagram illustrating calculation 
of fine offset level terms 18 used in the scene bad 
nonuniformity correction method 40 of the present 
55 invention that is employed in the scanning infrared 
sensor system 10 of Fig. 1 The following terms are 
defined and are used in implementing the method 
40 of the present invention. 
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- (m) B horizontal average of a vertical region 

F hp s high pass filter operator, 

hpi (m) b high pass version y,, 

T e threshold operator, _ 

b, (m) m threshold version of y,, 

B = segment boundary operator, 

F, p e low pass filter operator,^ 

Ipi (m) s low pass version of y t , and 

Ci (m) b vertical region correction term, 



■ 1/ 
L/ 



number of regions, and 
= samples/region. 



where 

I = {0, L-1}; L = 
r = {0, R-1}; R 

During an active field time, the scene based 
nonuniformity correction method 40 collects scene 
data and calculates the average within each region, 
for each line therein, illustrated in box 31. This 
operation is equivalent to implementing the expres- 
sion: 



70 



75 



20 



25 



Y,(m) is thus comprised of several column vec- 
tors, one for each region. These vectors are then 
high pass filtered (F hp ), illustrated in box 32, and 
thresholded (T), illustrated in box 33, to detect 
edges. The edges are marked as boundaries. 
Thus, 

hp, (m) = (F hp y ( ) 
b, (m) = (T hp,). 

These vectors are then low pass filtered, illus- 
trated in box 34, using the boundary information. 
Pixels marked as boundaries are ignored. This is 
denoted by the low pass filter operator, F lp and the 
boundary operator, B. That is, 

Ipi (m) = (F lp (B y,)). 

Next, each region vector, y, (m). is subtracted 
from its low pass filtered version in an adder 35, 
producing the correction term for each region, c, - 
(m). That is 

c, (m) = lp, (m) - y, (m). 

Finally, the correction terms are either applied 
individually for each region, or averaged together, 
wherein the boundary pixels are ignored. If they 
are averaged together, illustrated in box 36 the 
error estimate is calculated using the equation. 
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For the purposes of completeness, Fig. 3 is a 
flow diagram illustrating the scene based nonunifor- 
mity correction method 40 in accordance with the 
principles of the present invention. A video input 
signal is provided, indicated in step 41, such as 
from the infrared sensor 11 derived from an image. 
The video input signal is processed such that a 
vector representing offset correction terms is 
formed, and this vector is initially set to zero. Each 
element in this vector represents a correction term 
for a particular detector of the scanning infrared 
sensor 11. The vector is applied to each pixel of 
the image by the processor 13 as the pixels are 
read from the focal plane array 12. 

To measure the offset error, the image is sepa- 
rated into vertically oriented regions, each compris- 
ing a plurality of channels. The average of each 
channel within a region is computed, indicated in 
step 42, and a set of region vectors is formed, 
such that there is one region vector for each re- 
gion. Each region vector is then globally high pass 
filtered, and edges larger than a predefined thresh- 
old are detected, indicated in step 43, and marked, 
indicated in step 44. Then, each region vector is 
further separated into sub-regions, indicated in step 
45. The isolated sub-regions are high-pass filtered 
without regard to adjacent sub-regions, indicated in 
step 46. In a first embodiment of the method 40, 
the correction terms for each vertical region vector 
are averaged together, resulting in a single correc- 
tion vector, indicated in step 48. 

The correction terms calculated for each verti- 
cal region may also be applied individually to each 
the detectors. In this second embodiment, the off- 
set level error in each region for each channel is 
calculated, indicated in step 49, and wherein the 
offset level error at boundary edges is undefined. 
The correction terms corresponding to a region are 
applied as the detector 11 scans the scene and 
views a portion corresponding to that particular 
region. The correction terms are smoothed at re- 
gion boundaries to eliminate noise due to boundary 
transitions. This second method 40 is less sensitive 
to gain errors in the detector. 

Thus there has been described a new and 
improved scene based nonuniformity correction 
methods for use with imaging sensors. It is to be 
understood that the above-described embodiments 
are merely illustrative of some of the many specific 
embodiments which represent applications of the 
principles of the present invention. Clearly, nu- 
merous and other arrangements can be readily 
devised by those skilled in the art without departing 
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from the scope of the invention. 
Claims 

1. A scene based nonuniformity correction meth- 
od (40) for use with a scanning infrared sensor, 
said method (40) comprising the steps of: 

providing video input signal (41) derived 
from an image; 

processing the video input signal such that 
a vector representing offset correction terms if 
formed, wherein each element in the vector 
represents a correction term for a particular 
detector of the scanning infrared sensor (11), 
and wherein the vector is initially set to zero; 

measuring the offset error, by separating 
the image into vertically oriented regions, each 
comprising a plurality of channels; 

computing an average of each channel 
within a region (42); 

forming a set of region vectors, such that 
there is one region vector for each region; 

globally high-pass filtering each region 
vector wherein edges larger than a predefined 
threshold are detected (43). and marked (44); 

separating each region vector into sub- 
regions (45); 

locally high-pass filtering the isolated sub- 
regions without regard to adjacent sub-regions 
(46); 

computing (48, 49) the correction terms for 
each vertical region vector; and 

applying the calculated offset level errors 
individually to each detector of the sensor. 

2. The method of Claim 1 wherein the step of 
computing (48, 49) the correction terms com- 
prises the step of averaging (48) the correction 
terms for each vertical region vector together 
to produce a single correction vector. 



6. The method of Claim 1 wherein the step of 
locally high-pass filtering (46) comprises the 
step of median filtering the region vector and 
subtracting the resultant vector from an original 

5 region vector. 

7. The method of Claim 2 wherein the step of 
locally high-pass filtering (46) comprises the 
step of anti-median filtering the region vector. 

10 

8. The method of Claim 2 wherein the step of 
locally high-pass filtering (46) comprises the 
step of low-pass filtering the region vector and 
subtracting the resultant vector from an original 

75 region vector. 

9. The method of Claim 2 wherein the step of 
locally high-pass filtering (46) comprises the 
step of median filtering the region vector and 

20 subtracting the resultant vector from an original 

region vector. 

10. The method of Claim 3 wherein the step of 
locally high-pass filtering (46) comprises the 

25 step of anti-median filtering the region vector. 

11. The method of Claim 3 wherein the step of 
locally high-pass filtering (46) comprises the 
step of low-pass filtering the region vector and 

30 subtracting the resultant vector from an original 

region vector. 

12. The method of Claim 3 wherein the step of 
locally high-pass filtering (46) comprises the 

35 step of reedian filtering the region vector and 

subtracting the resultant vector from an original 
region vector. 



40 



3. The method of Claim 1 wherein the step of 
computing (48, 49) the correction terms com- 
prises the step of calculating (49) the offset 
level error in each region for each channel, 45 
wherein the offset level error at boundary 
edges are undefined. 

4. The method of Claim 1 wherein the step of 
locally high-pass filtering (46) comprises the so 
step of anti-median filtering the region vector. 

5. The method of Claim 1 wherein the step of 
locally high-pass filtering (46) comprises the 
step of low-pass filtering the region vector and 55 
subtracting the resultant vector from an original 
region vector. 
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Fig. 3 



PROVIDING A VIDEO INPUT SIGNAL 
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WITHIN EACH REGION. I, CALCULATE THE 
AVERAGE VALUE FOR EACH CHANNEL m 
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DETECT EDGES OF EACH REGION 



1 



MARK PIXELS WHOSE VALUES ARE OVER A THRESHOLD 
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USE MARKED PIXELS TO DIVIDE EACH REGION 
INTO INDIVIDUAL SUB-REGIONS 



45 

l 



HIGH-PASS FILTER THE ISOLATED SUB-REGIONS 
WITHOUT REGARD TO ADJACENT SUB-REGIONS 
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LOW PASS FILTER EACH SUB-REGION AND 
IGNORE EDGES OF THE SUB-REGIONS 
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CALCULATE THE OFFSET 
LEVEL ERROR IN EACH 

REGION FOR EACH 
CHANNEL. WHEREIN THE 
OFFSET LEVEL ERROR AT 
EDGES IS UNDEFINED 
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CALCULATE THE OFFSET 
LEVEL ERROR IN EACH 
REGION FOR EACH 
CHANNEL BY AVERAGING 
ACROSS THE REGIONS 
AND IGNORING 
UNDEFINED VALUES 
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